Open and Reproducible Science with R

font-family: ‘Helvetica’ width: 1024 height: 768

A two day workshop in 3 hours

Mine Çetinkaya-Rundel, Duke University
Colin Rundel, Duke University
François Michonneau, University of Florida
Tracy Teal, Data Carpentry

useR!2017 – Brussels
CC0

Reproducibility crisis

Nature Special Issue on Challenges in Irreproducible Research

Reproducibility matters

incremental: true

Lack of reproducibility in science causes significant issues

Science retracts gay marriage paper

Source: http://news.sciencemag.org/policy/2015/05/science-retracts-gay-marriage-paper-without-lead-author-s-consent

Reproducibility matters

Lack of reproducibility in science causes significant issues

Seizure study retracted after authors realize data got “terribly mixed”

From the authors of Low Dose Lidocaine for Refractory Seizures in Preterm Neonates (doi:10.1007/s12098-010-0331-7:

The article has been retracted at the request of the authors. After carefully re-examining the data presented in the article, they identified that data of two different hospitals got terribly mixed. The published results cannot be reproduced in accordance with scientific and clinical correctness.

Source: Retraction Watch

Reproducibility matters

Lack of reproducibility in science causes significant issues

Reproducibility = Accelerating science, including your own

B. Oliviera, Geeks and repetitive tasks Bruno Oliviera

Reproducible science accelerates scientific progress.

Computational research

incremental: true

Methods are codified by definition, yet still challenging to reproduce

See an experiment on reproducing reproducible computational research

Overcoming the training gap

Reproducible Science Curriculum Workshop & Hackathon Participants Reproducible Science Curriculum logo

Current Syllabus

Day 1 * Motivation of and introduction to Reproducible Research * Best practices for file naming and file organization * Best practices for tabular data * Literate programming and executable documentation of data modification

Day 2 * Version control and Git * Why automate? * Transforming repetitive R script code into R functions * Automated testing and integration testing * Sharing, publishing, and archiving for data and code

Exercise: Motivating reproducibility

type: titleonly

Exercise: Motivating reproducibility

type: prompt



This is a two-part exercise:

Part 1: Analyze + document

Part 2: Swap + discuss

Part 1: Analyze + document

type: prompt

Complete the following task and write instructions / documentation for your collaborator to reproduce your work starting with the original dataset (data/gapminder-5060.csv).

  1. Download material: http://bit.ly/2sEPe4z -> Full Link

  2. Visualize life expectancy over time for Canadians in the 1950s and 1960s using a line plot.

  3. Something should be clearly wrong with your plot, figure out (and document) what this is and come up with a fix that resolves this issue.

  4. Visualize life expectancy over time for Canadians again, with the corrected data.
    Stretch goal: Add additional lines for the life expectancy of Mexician and Americans as well.

Part 2: Swap + discuss

type: prompt

Introduce yourself to your collaborator (neighbor).

  1. Swap instructions / documentation with your collaborator. As you read it over think about how you would attempt to reproduce their work.

If your collaborator/neighbor does not have or is unfamiliar with the software you used we encourage you to given them a brief explination of what it is and why you chose it. (Remember, this could be part of the irreproducibility problem!)

  1. Then, talk to each other about challenges you faced (or didn’t face) or why you were or weren’t able to reproduce their work.

Reflection

type: prompt

This exercise: - What tools did you use (Excel, R / Python, Word / plain text etc.)? - What made it easy / hard for reproducing your partners’ work?

In a “real life” setting: - What would happen if your colleague/collaborator is no longer available to walk you through their analysis? - What would have to happen if you - had to swap out the dataset or extend the analysis? - caught further errors and had to re-create the analysis? - you had to revert back to the original dataset?

Four facets of reproducibility

type: titleonly

Four facets of reproducibility

  1. Documentation: difference between binary files (e.g. docx) and text files and why text files are preferred for documentation
    • markdown is a great format to document your workflow so that anyone can pick up your data and follow what you are doing
    • Use literate programming so that your analysis and your results are tightly connected

  2. Organization: tools to organize your projects so that you don’t have a single folder with hundreds of files

  3. Automation: the power of scripting to create automated data analyses

  4. Dissemination: publishing is not the end of your analysis, rather it is a way station towards your future research and the future research of others

Executable documentation

type: titleonly

Literate Programming

Provenance with results pasted into manuscript: Figure copy&pasted into MS Word

vs. Provenance of figures with Rmarkdown reports: Figure generating code following by generated figure

Demo - Literate Programming

type: titleonly

Executable data integrity checks


Life expectancy shouldn’t exceed even the most extreme age observed for humans.

if (any(gap_5060$lifeExp > 150)) {
  stop("improbably high life expectancies")
}
Error in eval(expr, envir, enclos): improbably high life expectancies

Executable data integrity checks


The library testthat allows us to make this a little more readable:

library(testthat)
expect_false(any(gap_5060$lifeExp > 150),
            "improbably high life expectancies")

Naming your files

type: titleonly

Research projects grow



- There are going to be files. Lots of files. - They will change over time. - They will have differing relationships to each other.

File organization and naming are effective weapons against chaos.

Three key principles for (file) names

Exercise

type: prompt



Your data files contain readings from a well plate, one file per well, using a specific assay run on a certain date, after a certain treatment.

Example:

$ ls *Plsmd*
2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_A01.csv
2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_A02.csv
2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_A03.csv
2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_B01.csv
2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_B02.csv
...
2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_H03.csv
> list.files(pattern = "Plsmd") %>% head
[1] 2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_A01.csv
[2] 2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_A02.csv
[3] 2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_A03.csv
[4] 2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_B01.csv
[5] 2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_B02.csv
[6] 2013-06-26_BRAFASSAY_Plsmd-CL56-1MutFrac_B03.csv

Example:

meta <- stringr::str_split_fixed(flist, "[_\\.]", 5)
colnames(meta) <- c("date", "assay", "experiment",
                    "well", "ext")
meta[,1:4]
     date         assay       experiment            well 
[1,] "2013-06-26" "BRAFASSAY" "Plsmd-CL56-1MutFrac" "A01"
[2,] "2013-06-26" "BRAFASSAY" "Plsmd-CL56-1MutFrac" "A02"
[3,] "2013-06-26" "BRAFASSAY" "Plsmd-CL56-1MutFrac" "A03"
[4,] "2013-06-26" "BRAFASSAY" "Plsmd-CL56-1MutFrac" "B01"
[5,] "2013-06-26" "BRAFASSAY" "Plsmd-CL56-1MutFrac" "B02"
[6,] "2013-06-26" "BRAFASSAY" "Plsmd-CL56-1MutFrac" "B03"

Organizing your files

type: titleonly

Best practices for file organization

class: centered-image

Noble (2009) A Quick Guide to Organizing Computational Biology Projects. PLoS Computational Biology 5 (7): e1000424

Noble, William Stafford. 2009. “A Quick Guide to Organizing Computational Biology Projects.” PLoS Computational Biology 5 (7): e1000424.

A data science project using R

title: false

|
+-- data-raw/
|   |
|   +-- gapminder-5060.csv
|   +-- gapminder-7080.csv.csv
|   +-- ....
|
+-- data-output/
|
+-- fig/
|
+-- R/
|   |
|   +-- figures.R
|     +-- data.R
|     +-- utils.R
|     +-- dependencies.R
|
+-- tests/
|
+-- manuscript.Rmd
+-- make.R

Some principles for file organization

Automation

type: titleonly

Write functions to automate everything

make_ms <- function() {
    rmarkdown::render("manuscript.Rmd",
                      "html_document")
    invisible(file.exists("manuscript.html"))
}
clean_ms <- function() {
    res <- file.remove("manuscript.html")
    invisible(res)
}
make_all <- function() {
    make_data()
    make_figures()
    make_tests()
    make_ms()
}
clean_all <- function() {
    clean_data()
    clean_figures()
    clean_ms()
}

Tests made easier

testthat includes a function called test_dir that will run tests included in files in a given directory. We can use it to run all the tests in our tests/ folder.

test_dir("tests/")

Let’s turn it into a function, so we’ll be able to add some additional functionalities to it a little later. We are also going to save it at the root of our working directory in the file called make.R:

## add this to make.R
make_tests <- function() {
    test_dir("tests/")
}

Version Control

type: titleonly

Version Control – bad

Run on file names

Version Control – OK

Use informatively named files

2013-10-14_manuscriptFish.doc
2013-10-30_manuscriptFish.doc
2013-11-05_manusctiptFish_intitialRyanEdits.doc
2013-11-10_manuscriptFish.doc
2013-11-11_manuscriptFish.doc
2013-11-15_manuscriptFish.doc
2013-11-30_manuscriptFish.doc
2013-12-01_manuscriptFish.doc
2013-12-02_manuscriptFish_PNASsubmitted.doc
2014-01-03_manuscriptFish_PLOSsubmitted.doc
2014-02-15_manuscriptFish_PLOSrevision.doc
2014-03-14_manuscriptFish_PLOSpublished.doc

Version Control – OK

Or zip the entire directory of your project files everytime you make a change, and save with date

Version Control – Best

Use a version control system (e.g. git)

Why use git and GitHub?

Why use Git?

Features of a hosting service like GitHub

Demonstration of RStudio’s git integration

type: titleonly

Dissemination - Sharing, publishing, archiving

type: titleonly

Why share / archive data & code?

incremental: true

increased visibility / citation

left: 70%

Piwowar & Vision (2013) Data reuse and the open data citation advantage.

Piwowar & Vision (2013) “Data reuse and the open data citation advantage.” PeerJ, e175

Figure 1: Citation density for papers with and without publicly available microarray data, by year of study publication.

Why share / archive data & code?

Better research

left: 70%

Wicherts et al (2011) Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results.

Wicherts et al (2011) “Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results.” PLoS ONE 6(11): e26828

Figure 1. Distribution of reporting errors per paper for papers from which data were shared and from which no data were shared.

Where archive & publish?

Where archive & publish?

R packages to help you

University libraries try to help

title: false

How to share, publish: file formats

Do’s * non-proprietary file formats * text file formats (.csv, .tsv, .txt)

Don’t’s * proprietary file formats (.xls) * data as PDFs or images * data in Word documents

Put a license on creative works

type: titleonly

Software licensing guide

left: 70%

Morin et al 2012, A Quick Guide to Software Licensing for the Scientist-Programmer

Morin, Andrew, Jennifer Urban, and Piotr Sliz. 2012. “A Quick Guide to Software Licensing for the Scientist-Programmer.” PLoS Computational Biology 8 (7): e1002598.

Don’t put a license on facts (a.k.a. data)

type: titleonly

Licenses versus community norms

CC Zero

From the Panton Principles: > [In] the scholarly research community the act of citation is a commonly held community norm when reusing another community member’s work. […] A well functioning community supports its members in their application of norms, whereas licences can only be enforced through court action and thus invite people to ignore them when they are confident that this is unlikely.

Good - Better - Best



Peng, R. D. (2011) Reproducible Research in Computational Science Peng, R. D. “Reproducible Research in Computational Science” Science 334, no. 6060 (2011): 1226–1227

Forming reproducible habits pays off

incremental: true

Acknowledgements

Eating our dogfood: text formats, version control, sharing

This slideshow was generated as HTML from Markdown using RStudio.

The Markdown sources, and the HTML, are hosted on Github: https://github.com/fmichonneau/2017-useR-reproducibility

Rstudio screenshot of this presentation RStudio Logo

Further Resources